Normalizing counts of plant-parts-of-interest

ABSTRACT

Implementations are described herein for normalizing counts of plant-parts-of-interest detected in digital imagery to account for differences in spatial dimensions of plants, particularly plant heights. In various implementations, one or more digital images depicting a top of a first plant may be processed. The one or more digital images may have been acquired by a vision sensor carried over top of the first plant by a ground-based vehicle. Based on the processing: a distance of the vision sensor to the first plant may be estimated, and a count of visible plant-parts-of-interest that were captured within a field of view of the vision sensor may be determined. Based on the estimated distance, the count of visible plant-parts-of-interest may be normalized with another count of visible plant-parts-of-interest determined from one or more digital images capturing a second plant.

BACKGROUND

Crops may have various constituent parts, referred to herein as“plant-parts-of-interest,” that are of nutritional and/or economicinterest to various entities. Plant-parts-of-interest may include, forinstance, flowers, buds, pods (e.g., for beans, peas), leaves, stems,berries, fruit, etc. Plant-parts-of-interest of crops may be counted forvarious reasons, such as crop yield prediction, varietal selection,breeding, plant disease diagnosis, general agricultural planning, etc. Avariety of different techniques exist for countingplant-parts-of-interest on crops. Perhaps the simplest techniqueinvolves a human manually counting plant-parts-of-interest on a subsetof plants, and then extrapolating that count across a superset of plants(e.g., an entire plot or field). However, with many crops, there may betoo many plant-parts-of-interest for a human to easily count.

SUMMARY

Computer vision-based techniques may be better suited to count massivenumbers of plant-parts-of-interest than manual counting by humans, butthose counts may be limited to those plant-parts-of-interest that arevisible in a field of view (FOV) of a vision sensor. Occludedplant-parts-of-interest may need to be extrapolated from the visibleplant-parts-of-interest, which may be difficult and/or inaccuratewithout scalable techniques for determining spatial dimensions of aplant. Additionally, while the height of the vision sensor may remainmore-or-less constant, different plants will have different spatialdimensions, particularly heights, depending on a variety of factors,such as time since planting, plant breed, agricultural management (e.g.,irrigation, application of chemicals such as pesticides, herbicides,fertilizer, etc.), and so forth.

Accordingly, implementations are described herein for normalizing countsof plant-parts-of-interest detected in digital imagery to account fordifferences in spatial dimensions of plants, particularly plant heights.In various implementations, a depth or range-capable vision sensor suchas a stereoscopic camera or RGBd camera may be carried over top of aplurality of plants, e.g., by a farm vehicle such as a tractor or anagricultural robot, to obtain a plurality of digital images that includedepth or range information (“depth” and “range” will be used hereininterchangeably).

As noted above, different plants will have different heights. Forexample, one plot of a farm may be growing a first varietal of soybeanand another plot of the farm may be growing a second varietal of soybeanthat is shorter than the first varietal. Assuming the vision sensor iscarried over these two varietals at a constant height, the secondvarietals will be farther from the vision sensor than the firstvarietals. Even if both varietals are producing plant-parts-of-interestat similar densities, greater numbers of plant-parts-of-interest (e.g.,soybean pods) will be visible in digital images depicting the secondvarietals. To account for these different heights and mitigate againstunder or over predicting counts of plant-parts-of-interest, distances ofthe respective varietals from the vision sensor (“ranges-to-canopies”herein) may be estimated and used to normalize or calibrate counts ofvisible plant-parts-of-interest. For example, in some implementations, acount of visible plant-parts-of-interest in a digital image may bedivided by the distance between the vision sensor and top(s) of plant(s)in the digital image.

A range-to-canopy may be determined in various ways. As noted above, thevision sensor may be range capable, and therefore, the vision data itgenerates may include range data, such as pixel-wise range values.However, a “top” of a plant may not be readily apparent, especially fromoverhead, because the plant likely has a multitude of components such asleaves, stalks, etc., that have a corresponding distribution of heights.In various implementations, this distribution may be captured in adistribution of pixel-wise range values and used to estimate arange-to-canopy. For example, some quantile of the distribution, such asthe top 10% most frequent pixel-wise range values, the closest 5%, theaverage of the top 10%, etc., may be used as an estimate of the distancebetween the plant and the vision sensor.

Visible plant-parts-of-interest such as pods, flowers, buds, fruit,berries, etc., may be counted in digital imagery using a variety oftechniques. In some implementations, a deep learning convolutionalneural network (CNN) may be trained to detect plant-parts-of-interest,e.g., using training digital images in which targetplant-parts-of-interest are annotated with bounding boxes or pixel-wiseannotations. In various implementations, the model may be a deep objectdetection model and/or a deep segmentation model.

Normalized counts of visible plant-parts-of-interest may be used for avariety of purposes. In some implementations, additionalplant-parts-of-interest that are not visible in the digital imagery,e.g., because they are occluded, may be extrapolated from the normalizedcounts of visible plant-parts-of-interest so that a total count ofplant-parts-of-interest can be predicted. In some such implementations,other signals may be used in addition to visible plant-parts-of-interestto extrapolate these non-visible plant-parts-of-interest. These othersignals may include, for instance, color (which can indicate health of aplant), climate data, agricultural management data, prior counts ofrelated plant-parts-of-interest (e.g., previous flower counts may beused as a basis for predicting subsequent pod counts), other spatialdimensions of plants (e.g., height, width, diameter), canopy density,etc.

In some implementations, total counts of plant-parts-of-interest may beestimated at various stages of growth of a crop and used to project acrop yield. In some implementations, a time-series machine learningmodel such as a recurrent neural network (RNN) may be trained to processsequences of estimated total counts of plant-part-of-interest obtainedat different stages of a crop cycle in order to project a crop yield.

In some implementations, a method may be implemented using one or moreprocessors and may include: processing one or more digital imagesdepicting a top of a first plant, wherein the one or more digital imagesdepicting the top of the first plant are acquired by a vision sensorcarried over top of the first plant by a ground-based vehicle; based onthe processing: estimating a distance of the vision sensor to the firstplant, estimating a height of the first plant, and determining a countof visible plant-parts-of-interest that were captured within a field ofview of the vision sensor; based on the estimated distance, normalizingthe count of visible plant-parts-of-interest with another count ofvisible plant-parts-of-interest determined from one or more digitalimages capturing a second plant; and predicting a crop yield based onthe normalized count of visible plant-parts-of-interest and the heightof the first plant.

In various implementations, the estimated distance of the vision sensorto the first plant may be estimated based on a distribution ofpixel-wise range values of one or more of the digital images capturingthe first plant. In various implementations, the estimated distance ofthe vision sensor to the first plant may be calculated as a quantile ofthe distribution of pixel-wise range values.

In various implementations, the count of visible plant-parts-of-interestmay be determined using a convolutional neural network. In variousimplementations, the plant-parts-of-interest may be bean pods. Invarious implementations, the estimated distance between the visionsensor and the first plant may be determined based on the estimatedheight of the first plant and a height of the vision sensor. In variousimplementations, the first plant and second plants may be differentvarietals of soybean plants.

In addition, some implementations include one or more processors (e.g.,central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or morecomputing devices, where the one or more processors are operable toexecute instructions stored in associated memory, and where theinstructions are configured to cause performance of any of theaforementioned methods. Some implementations also include one or morenon-transitory computer readable storage media storing computerinstructions executable by one or more processors to perform any of theaforementioned methods. Yet other implementations include agriculturalvehicles, such as robots, that are equipped with edge processor(s)configured to carry out selected aspects of the present disclosure.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically depicts an example environment in which disclosedtechniques may be employed in accordance with various implementations.

FIG. 2 depicts an example of how computing nodes configured withselected aspects of the present disclosure may be deployed in a field.

FIG. 3A and FIG. 3B schematically depict an example of how adistribution of ranges-to-canopies may be determined and used toestimate one range-to-canopy for purposes of normalization.

FIG. 4 depicts an example of how various components described herein mayexchange and process data to normalize counts of plant-parts-of-interestand make downstream inferences therefrom.

FIG. 5 is a flowchart of an example method in accordance with variousimplementations described herein.

FIG. 6 schematically depicts an example architecture of a computersystem.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates an environment in which one or moreselected aspects of the present disclosure may be implemented, inaccordance with various implementations. The example environmentincludes an agricultural information system 104, one or more clientdevices 106 _(1-X), and human-controlled and/or autonomous farm vehicles107 ₁₋₂ that can be operated to carry any number of vision sensors 108_(1-N) over top plants of one or more fields 112. The various componentsdepicted in FIG. 1 may be in network communication with each other viaone or more networks 110, such as one or more wide area networks(“WANs”) such as the Internet, and/or via one or more local areanetworks (“LANs”, e.g., Wi-Fi, Ethernet, various mesh networks) and/orpersonal area networks (“PANs”, e.g., Bluetooth). Field(s) 112 may beused to grow various types of crops that may produceplant-parts-of-interest, where the interest may be economic and/ornutritional, for instance. These crops may include but are not limitedto strawberries, tomato plants, soy beans, other types of beans, corn,lettuce, spinach, beans, cherries, nuts, cereal grains, berries, grapes,and so forth.

An individual (which in the current context may also be referred to as a“user”) may operate a client device 106 to interact with othercomponents depicted in FIG. 1 . Each client device 106 may be, forexample, a desktop computing device, a laptop computing device, a tabletcomputing device, a mobile phone computing device, a computing device ofa vehicle of the participant (e.g., an in-vehicle communications system,an in-vehicle entertainment system, an in-vehicle navigation system), astandalone interactive speaker (with or without a display), or awearable apparatus that includes a computing device, such as ahead-mounted display (“HMD”) that provides an AR or VR immersivecomputing experience, a “smart” watch, and so forth. Additional and/oralternative client devices may be provided.

Each of client devices 106 and/or agricultural information system 104may include one or more memories for storage of data and softwareapplications, one or more processors for accessing data and executingapplications, and other components that facilitate communication over anetwork. In various implementations, some vision sensors 108, such asvision sensor 108 ₁ associated with aerial drone 107 ₁ and/or visionsensors 108 _(2-N) mounted to a boom 130 of tractor 107 ₂, may beintegrated into a computing node (which may or may not be modular and/orremovable from the vehicle 107 that carries it) that also includes logicsuch as processor(s), application-specific integrated circuits (ASICs),field-programmable gate arrays (FPGA), etc. FIG. 2 schematically depictsone example of such a vision-sensor-equipped computing node, and will bediscussed in more detail shortly.

Vision sensors 108 _(1-N) may take various forms, particularly formsthat are capable of detecting depth or range (“depth” and “range” willbe used herein interchangeably). In some implementations, a visionsensor 108 may be a stereoscope camera, and/or may include multiple 2Dcameras that are operated in cooperation as a stereoscopic visionsensor. In some implementations, a single camera may be operated as a defacto stereoscopic camera by capturing two images in succession fromslightly different angles (e.g., as the vehicle 107 carrying the cameramoves) and processing them using stereoscopic techniques. Additionallyor alternatively, in some implementations, one or more vision sensors108 may take the form of a range-capable sensor such as a lightdetection and ranging (LIDAR) sensor.

Techniques described herein may be performed in whole or in part byvarious components depicted in FIG. 1 . For example, aspect(s) ofagricultural information system 104 may be implemented in whole or inpart on client device(s) 106, agricultural information system 104,and/or by the computing node(s) mentioned previously.

Each client device 106, may operate a variety of different applicationsthat may be used, for instance, to obtain and/or analyze variousagricultural inferences that were generated using techniques describedherein. For example, a first client device 106 ₁ operates agricultural(AG) client 107 (e.g., which may be standalone or part of anotherapplication, such as part of a web browser). Another client device 106 xmay take the form of a HMD that is configured to render 2D and/or 3Ddata to a wearer as part of a VR immersive computing experience. Forexample, the wearer of client device 106 x may be presented with 3Dpoint clouds representing various plant-parts-of-interest, such asfruits of crops, weeds, crop yield predictions, etc. The wearer mayinteract with the presented data, e.g., using HMD input techniques suchas gaze directions, blinks, etc.

Individual farm vehicles 107 may take various forms. As shown in FIG. 1and mentioned previously, some farm vehicles may be operated at leastpartially autonomously, and may include, for instance, unmanned aerialvehicle 107 ₁ that carries a vision sensor 108 ₁ that acquires visionsensor data such as digital images from overhead field(s) 112. Otherautonomous farm vehicles (e.g., robots) not depicted in FIG. 1 mayinclude a robot that is propelled along a wire, track, rail or othersimilar component that passes over and/or between crops, a wheeledrobot, or any other form of robot capable of being propelled orpropelling itself past/through/over crops of interest. In someimplementations, different autonomous farm vehicles may have differentroles, e.g., depending on their capabilities. For example, in someimplementations, one or more robots may be designed to acquire data,other robots may be designed to manipulate plants or perform physicalagricultural tasks, and/or other robots may do both. Other farmvehicles, such as a tractor 107 ₂, may be autonomous, semi-autonomous,and/or human-driven. As noted above, any of farm vehicle 107 may beequipped with various types of sensors, such as vision sensors 108_(1-N). Farm vehicle 107 may be equipped with other sensors as well,such as inertial measurement unit (IMU) sensors, Global PositioningSystem (GPS) sensors, X-ray sensors, moisture sensors, barometers (forlocal weather information), photodiodes (e.g., for sunlight),thermometers, etc.

In various implementations, agricultural information system 104 mayinclude a counting module 114, a range module 116, a normalizationmodule 118, and an inference module 120. Agricultural information system104 may also include one or more databases 115, 121 for storing variousdata used by and/or generated by modules 114-120, such as data gatheredby sensors carried by farm vehicles 107, agricultural inferences,machine learning models that are applied and/or trained using techniquesdescribed herein to generate agricultural inferences, and so forth. Insome implementations one or more of modules 114-120 may be omitted,combined, and/or implemented in a component that is separate fromagricultural information system 104. In various implementations,agricultural information system 104 may be implemented across one ormore computing systems that may be referred to as the “cloud.”

Counting module 114 may be configured to process digital images acquiredby vision sensors 108 _(1-N) to determine counts of visibleplant-parts-of-interest that were within fields of view of visionsensors 108 _(1-N) when the digital images were acquired. Countingmodule 114 may employ various techniques to count visibleplant-parts-of-interest. In some implementations, counting module 114may determine counts of plant-parts-of-interest using one or moremachine learning models stored in database 115. A machine learning modelthat is used in such a context may take various forms, including but notlimited to a convolutional neural network (CNN).

In some implementations, a machine learning model employed by countingmodule 114 may be trained to perform object recognition, in which caseits output may be indicative of bounding shapes such as bounding boxes.Additionally or alternatively, in some implementations, such a machinelearning model may be trained to perform image segmentation, in whichcase its output may be pixel-wise annotations (orpixel-region-annotations). Other, segmentation and/or object recognitiontechniques that may or may not be implemented using artificialintelligence, such as thresholding, clustering, compression-basedmethods, histogram-based methods, region-growing methods, partialdifferential equation-based methods, graph partitioning methods,watershed methods, and so forth, are also contemplated.

As noted previously, the count of plant-parts-of-interest that arevisible in a given digital image may depend on, in addition to theactual density of the plant-parts-of-interest, a distance between thevision sensor 108 and the plant. If the vision sensor is relatively faraway from the plant, e.g., because the plant is relatively short, then arelatively large number of plant-parts-of-interest may be captured inthe vision sensor's FOV. By contrast, if the vision sensor is relativelyclose to the plant, e.g., because the plant is relatively tall, then arelatively small number of plant-parts-of-interest may be captured inthe vision sensor's FOV.

Accordingly, range module 116 and normalization module 118 (which may becombined in a single module in some cases) may be configured tonormalize counts generated by counting module 114 to account and/ormitigate for disparities in plant spatial dimensions, including but notlimited to plant height. For example, range module 116 may be configuredto process range data to estimate a distance (referred to herein as a“range-to-canopy”) between a vision sensor 108 and top(s) of plant(s).In some implementations, these range data may be integral with visionsensor data captured by a vision sensor 108, e.g., in the form ofpixel-wise range values. Range module 116 may additionally be configuredto process range data to identify ranges other than ranges-to-canopies.For example, in some implementations, range module 116 may process rangedata indicative of a distance between the ground and the vision sensor(referred to herein as “range-to-ground”). In some such implementations,a height of a plant may be determined based on a difference betweenrange-to-ground and a range-to-canopy.

Based on the various range(s) provided by range module 116 (e.g.,range-to-ground, range-to-canopy), normalization module 118 maynormalize counts of visible plant-parts-of-interest generated bycounting module 114 with other counts of visible plant-parts-of-interestgenerated by counting module 114 based on other plants (e.g.,neighboring plants, different varietals, different plot or field ofplants, different greenhouse, etc.). For example, in someimplementations, a count generated by counting module 114 for eachinstance of captured vision data (e.g., each digital image) may bedivided by the range-to-canopy calculated for the same instance ofcaptured vision data. In other implementations, the raw count may benormalized in other ways. In some implementations, the raw count may bedivided by a power (other than one) of the range-to-canopy. Forinstance, if the range-to-canopy is x meters, then the raw count couldbe divided by the second power of x (x²), or even a non-integer power(e.g., x^(1.5), x^(2.5), etc.).

Based on the normalized counts generated by normalization module 118, aswell as on any number of other inputs, inference module 120 may beconfigured to make a variety of different agricultural inferences. Forexample, inference module 120 may process time series data that includesnormalized counts of plant-parts-of-interest based on one or moretime-series machine learning models stored in database 121 to generateoutput indicative of predicted crop yield. Other inputs (e.g., thatcorrespond temporally, or that are preprocessed to correspondingtemporally, with normalized counts of plant-parts-of-interest) that maybe used by inference module 120 to make agricultural inferences mayinclude, but are not limited to, satellite imagery, climate data (sensedlocally or obtained from remote databases), agricultural management data(e.g., applied chemicals, applied irrigation, etc.), soil measurements,prior counts of precursor plant-parts-of-interest (e.g., flowers thateventually morph into other plant-parts-of-interest), and so forth.

FIG. 2 depicts an overhead view of an example field of plants 240 ₁₋₁₂.Boom 130 mounted to tractor 107 ₂ (mostly not visible in FIG. 2 , seeFIG. 1 ) is being carried over plants 240 ₁₋₁₂ as shown by the arrow togather sensor data. Boom 130 may include, for instance, sprinklers forirrigation, sprayers for chemical application, etc. Also mounted on boom130 are a plurality of modular computing nodes 234 _(1-M) that areconfigured with selected aspects of the present disclosure. Althoughshown as boxes on top of boom 130 in FIG. 2 , modular computing nodes234 _(1-M) may alternatively be mounted at other locations of boom 130,such as on its sides or bottom. And while multiple modular computingnodes 234 _(1-M) are depicted in FIG. 2 , any number of modularcomputing nodes 234, such as a single modular computing node 234, may bedeployed in similar fashions.

As shown by the called-out window at top right, modular computing node234 _(M) includes one or more vision sensors 108 _(1-N), one or morelights 238, a light controller 241, and logic 242 that is configured tocarry out selected aspects of the present disclosure. Other modularcomputing nodes 234 _(1-(M-1)) may or may not be similarly configured.Vision sensors 108 _(1-N) may take various forms of range-capable visionsensors described previously, and may or may not be homogenous.

Light(s) 238 and light controller 241 may be configured to illuminateplants 240, e.g., in synch with operation of vision sensors 108 _(1-N),in order to make sure that the vision data that is captured isilluminated sufficiently so that it can be used to make accurateagricultural inferences. Light(s) 238 may take various forms, such asthe light emitting diode (LED) depicted in FIG. 2 , halogen lamps,incandescent lamps, etc. In various implementations, light(s) 238 may beoperated, e.g., by light controller 241, to emit various amounts and/orstrengths of light (or more generally, electromagnetic radiation).

Modular computing node 234 _(M) also includes one or more wirelessantenna 244 _(1-P). In some implementations, each wireless antenna 244may be configured to transmit and/or receive different types of wirelessdata. For example, a first antenna 244 ₁ may be configured to transmitand/or receive Global Navigation Satellite System (GNSS) wireless data,e.g., for purposes such as localization and/or ROI establishment.Another antenna 244 _(P) may be configured to transmit and/or receiveIEEE 802.12 family of protocols (Wi-Fi) or Long-Term Evolution (LTE)data. Another antenna 244 may be configured to transmit and/or receive5G data. Any number of antennas 244 may be provided to accommodate anynumber of wireless technologies.

In some implementations, a modular computing node 234 may be capable oflocalizing itself within agricultural field 112 using varioustechnologies. For example, the GNSS antenna 244 ₁ may interact withsatellite(s) to obtain a position coordinate. Additionally oralternatively, modular computing node 234 may use techniques such asinertial measurement units (IMU) that are generated by, for instance,sensor(s) integral with wheels (not depicted) of tractor 107 ₂,accelerometer(s), gyroscope(s), magnetometer(s), etc. In yet otherimplementations, wireless triangulation may be employed.

Logic 242 may include various types of circuitry (e.g., processor(s),FPGA, ASIC) that is configured to carry out selected aspects of thepresent disclosure. For example, and as shown in the called-out windowat top left in FIG. 2 , logic 242 may include any number of tensorprocessing units (TPU) 246 _(1-Q), a storage module 248, and a stereomodule 250 (one or more graphical process units (GPU) and/or centralprocessing units (CPU) may also be present, even if not depicted). Otherconfigurations are possible. For example, instead of some number ofTPUs, in some examples, a modular computing node 234 may include somenumber of GPUs, each with some number of cores. With the exampleoperational parameters of modular computing node 234 described herein,in some examples, modular computing node 234 may be capable of beingmoved (or moving itself) at various speeds to perform its tasks (e.g.,make agricultural inferences).

Storage module 248 may be configured to acquire and store, e.g., invarious types of memories onboard modular computing node 234, sensordata acquired from one or more sensors (e.g., vision sensors 108_(1-N)). Stereo module 250 may be provided in some implementations inorder to reconcile images captured by 2D vision sensors that areslightly offset from each other, and/or to generate 3D images and/orimages with depth/range. In various implementations, logic (e.g., 242)of modular computing node(s) 234 _(1-M) may perform, separately or incooperation with each other, selected aspects of the present disclosure,including aspects of agricultural information system 104, such ascounting module 114, range module 116, normalization module 118, and/orinference module 120.

FIGS. 3A and 3B demonstrate an example of how the boom-mountedarrangement of modular computing nodes 234 _(1-N) of FIG. 2 and theirintegral vision sensors (not depicted in FIG. 3 , see FIGS. 1 and 2 )may be operated. As shown, modular computing nodes 234 _(1-M) areseparated from the tops (or canopies) of plants 340 ₁₋₄ generally by adistance (range-to-canopy) that is indicated as D in FIG. 3A. It is alsoevident that each plant 340 has multiple upwardly extending components,and that plants 340 ₁₋₄ do not have uniform heights, individually and/orrelative to each other.

Vision sensors (again, not depicted in FIG. 3A, see FIGS. 1 and 2 ) ofmodular computing nodes 234 _(1-M) may generate range data as describedpreviously, e.g., from pixel-wise range values contained in digitalimage(s). This range data may be analyzed by range module 116 todetermine a distribution 352 of ranges-to-canopies between modularcomputing nodes 234 _(1-M) and plants 340 ₁₋₄. The distribution isdepicted in FIG. 3A as slightly elevated above the tops of plants 340₁₋₄ for illustrative purposes.

As shown in FIG. 3B, normalization module 118 may process distribution352 of heights in order to, for instance, generate a histogram 354 ofheights. Normalization module 118 may then estimate a height across oneor more of plants 340 ₁₋₄ by, for instance, selecting a quantile of thedistribution's histogram 354. For example, one option would be quantile356A, which may represent an average or median of all ranges. Anotheroption would be the quantile range 356B, which may represent, forinstance, the top 10% most frequently-occurring pixel-wise range values.For example, an average of the top 10% most frequently-occurringpixel-wise range values could be used as the estimated range-to-canopy,or in FIG. 3A, D. Similar techniques may be employed in order todetermine a distance (or range-to-ground) between vision sensors andground 351 in FIG. 3A (although the distribution of ground ranges likelywould be more uniform than the tops of plants 340 ₁₋₄). In someimplementations, a difference between range-to-canopy andrange-to-ground may be used as estimated height(s) of plant(s) 340 ₁₋₄.

While not of entirely uniform height, plants 340 ₁₋₄ in FIG. 3A arerelatively close to each other in height, which may be expected if theyare the same varietal grown under similar conditions (as likely would bethe case when grown next to each other as shown). Thus, whiledistances-to-canopies can be estimated on an individual plant basis, asingle range-to-canopy can alternatively be estimated across multipleplants having more or less the same general heights.

For example, one farm may grow multiple varietals of a particular plantfor a variety of reasons, such as satisfying demand for multiplevarietals, being able to select between varietals depending on results,risk diversification, etc. If these different varietals have differentheights, then in some implementations, a range-to-canopy may beestimated for multiple plants of one varietal, and anotherrange-to-canopy may be estimated for multiple plants of anothervarietal. Normalization module 118 may then normalize counts acrossthese multiple ranges-to-canopies to account for the disparity inheights between varietals. Assuming the multiple different varietalsgenerate plant-parts-of-interest at similar densities, thisnormalization will prevent shorter plants from being interpreted as moredensely producing.

FIG. 4 depicts an example of how various components described herein mayexchange and process data to normalize counts of plant-parts-of-interestand make downstream inferences therefrom. A sequence of digital images460 that depict plants from overhead is generated, e.g., by one or morevision sensors 108 carried by a vehicle such as a land-based vehicle(e.g., tractor 107 ₂) or an aerial drone (e.g., 107 ₁). These images 460are provided to counting module 114 and range module 116 for processing.As described previously, counting module 114 may employ varioustechniques, such as machine-learning segmentation and/or objectrecognition (e.g., using a model from database 115), to generate a rawcount of visible plant-parts-of-interest in each digital image.Meanwhile (e.g., in parallel), range module 116 may process the images460 to generate range-to-canopy and/or range-to-ground distributionsand/or estimated values for each image, for groups of images (e.g.,groups of images depicting the same varietal), etc.

Normalization module 118 may use the range distributions/estimatesreceived from range module 116 to normalize the raw counts ofplant-parts-of-interest received from counting module 114. Normalizationmodule 118 may then provide the normalized count(s) to inference module120. Inference module 120 may apply various types of machine learningmodels from database 121 to various data, such as digital images 460,climate data, agricultural management data, soil data, as well as thenormalized counts received from normalization module 118, to generatevarious inferences 462. These inference 462 may include, but are notlimited to, crop yield predictions, plant disease diagnosis,agricultural recommendations (e.g., more irrigation, less pesticide,etc.), crop rotation recommendations, soil organic compound (SOC)estimates, etc.

FIG. 5 illustrates a flowchart of an example method 500 for practicingselected aspects of the present disclosure. For convenience, operationsof method 500 will be described as being performed by a systemconfigured with selected aspects of the present disclosure. Otherimplementations may include additional operations than those illustratedin FIG. 5 , may perform operation(s) of FIG. 5 in a different orderand/or in parallel, and/or may omit one or more of the operations ofFIG. 5 .

At block 502, the system, e.g., by way of counting module 114 and/orrange module 116, may process one or more digital images depicting thetop of a first plant. In various implementations, the one or moredigital images may have been acquired by a vision sensor (e.g., 108)carried over top of the first plant by a ground-based vehicle (e.g.,tractor 107 ₂).

As indicated by the narrowing of the boxes in FIG. 5 , operations504-508 may be performed as part of/based on the processing of block502. At block 504, the system, e.g., by way of range module 116, mayestimate a distance of the vision sensor to the first plant(range-to-canopy described previously). At block 506, the system, e.g.,by way of range module 116, may estimate a height of the first plant(range-to-ground described previously). At block 506, the system, e.g.,by way of counting module 114, may determine a count (e.g., the rawcount of FIG. 4 ) of visible plant-parts-of-interest that were capturedwithin a field of view of the vision sensor.

Based on the distance estimated at block 504, at block 510, the system,e.g., by way of normalization module 118, may normalize the count ofvisible plant-parts-of-interest with another count of visibleplant-parts-of-interest determined from one or more digital imagescapturing a second plant. At block 512, the system, e.g., by way ofinference module 120, may predict a crop yield based on the normalizedcount of visible plant-parts-of-interest generated at block 510 and theheight of the first plant estimated at block 506.

Other applications of techniques described herein are also contemplated.For example, range-to-canopy can be used to calculate the sizes ofplant-parts-of-interest, such as the size of soybean pods. This soybeanpod size could further be used to predict crop yield. As anotherexample, range-to-canopy and/or range-to-ground could be used todetermine other spatial dimensions of a plant, such as its width. Theplant's width and height may be used, e.g., by counting module 114, toextrapolate a total count of plant-parts-of-interest, for instance.

FIG. 6 is a block diagram of an example computing device 610 that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein. Computing device 610 typically includes at least oneprocessor 614 which communicates with a number of peripheral devices viabus subsystem 612. These peripheral devices may include a storagesubsystem 624, including, for example, a memory subsystem 625 and a filestorage subsystem 626, user interface output devices 620, user interfaceinput devices 622, and a network interface subsystem 616. The input andoutput devices allow user interaction with computing device 610. Networkinterface subsystem 616 provides an interface to outside networks and iscoupled to corresponding interface devices in other computing devices.

User interface input devices 622 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In some implementations in which computingdevice 610 takes the form of a HMD or smart glasses, a pose of a user'seyes may be tracked for use, e.g., alone or in combination with otherstimuli (e.g., blinking, pressing a button, etc.), as user input. Ingeneral, use of the term “input device” is intended to include allpossible types of devices and ways to input information into computingdevice 610 or onto a communication network.

User interface output devices 620 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, one or more displays forming part of a HMD, or some othermechanism for creating a visible image. The display subsystem may alsoprovide non-visual display such as via audio output devices. In general,use of the term “output device” is intended to include all possibletypes of devices and ways to output information from computing device610 to the user or to another machine or computing device.

Storage subsystem 624 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 624 may include the logic toperform selected aspects of the method 500 described herein, as well asto implement various components depicted in FIGS. 1, 2, and 4 .

These software modules are generally executed by processor 614 alone orin combination with other processors. Memory subsystem 625 used in thestorage subsystem 624 can include a number of memories including a mainrandom access memory (RAM) 630 for storage of instructions and dataduring program execution and a read only memory (ROM) 632 in which fixedinstructions are stored. A file storage subsystem 626 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 626 in the storage subsystem 624, or inother machines accessible by the processor(s) 614.

Bus subsystem 612 provides a mechanism for letting the variouscomponents and subsystems of computing device 610 communicate with eachother as intended. Although bus subsystem 612 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 610 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 610depicted in FIG. 6 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 610 are possible having more or fewer components thanthe computing device depicted in FIG. 6 .

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

What is claimed is:
 1. A method implemented using one or moreprocessors, comprising: obtaining a plurality of digital images acquiredby a range-capable vision sensor carried by a ground-based vehicle pastat least first and second plants, wherein differing spatial dimensionsof the first and second plants cause parts of the first and secondplants to be first and second distances, respectively, from therange-capable vision sensor as it passes the at least first and secondplants; processing one or more of the digital images depicting the firstplant; based on the processing: estimating the first distance of therange-capable vision sensor to the first plant, estimating a spatialdimension of the first plant, and determining a count of visibleplant-parts-of-interest of the first plant that were captured within afield of view of the range-capable vision sensor; based on the estimatedfirst distance, normalizing the count of visible plant-parts-of-interestof the first plant with another count of visible plant-parts-of-interestdetermined from one or more of the digital images depicting the secondplant; and predicting a crop yield using the normalized count of visibleplant-parts-of-interest and the first spatial dimension of the firstplant.
 2. The method of claim 1, wherein the estimated first distance ofthe range-capable vision sensor to the first plant is estimated based ona distribution of pixel-wise ranges between the range-capable visionsensor and the first plant, wherein the pixel-wise ranges are detectedby the range-capable vision sensor and included in one or more of thedigital images capturing the first plant.
 3. The method of claim 2,wherein the estimated first distance of the range-capable vision sensorto the first plant is calculated as a quantile of the distribution ofpixel-wise ranges.
 4. The method of claim 1, wherein the count ofvisible plant-parts-of-interest is determined using a convolutionalneural network.
 5. The method of claim 1, wherein the visibleplant-parts-of-interest comprise bean pods.
 6. The method of claim 1,wherein the first plant and second plants are different varietals ofsoybean plants.
 7. The method of claim 1, wherein the spatial dimensionof the first plant comprises a height of the first plant.
 8. A systemcomprising one or more processors and memory storing instructions that,in response to execution by the one or more processors, cause the one ormore processors to: obtain a plurality of digital images acquired by arange-capable vision sensor carried by a ground-based vehicle past atleast first and second plants, wherein differing spatial dimensions ofthe first and second plants cause parts of the first and second plantsto be first and second distances, respectively, from the range-capablevision sensor as it passes the at least first and second plants; processone or more of the digital images depicting the first plant; based onthe processing: estimate the first distance of the range-capable visionsensor to the first plant, estimate a spatial dimension of the firstplant, and determine a count of visible plant-parts-of-interest of thefirst plant that were captured within a field of view of therange-capable vision sensor; based on the estimated first distance,normalize the count of visible plant-parts-of-interest of the firstplant with another count of visible plant-parts-of-interest determinedfrom one or more of the digital images depicting the second plant; andpredict a crop yield using the normalized count of visibleplant-parts-of-interest and the first spatial dimension of the firstplant.
 9. The system of claim 8, wherein the estimated first distance ofthe range-capable vision sensor to the first plant is estimated based ona distribution of pixel-wise ranges between the range-capable visionsensor and the first plant, wherein the pixel-wise ranges are detectedby the range-capable vision sensor and included in one or more of thedigital images capturing the first plant.
 10. The system of claim 9,wherein the estimated first distance of the range-capable vision sensorto the first plant is calculated as a quantile of the distribution ofpixel-wise ranges.
 11. The system of claim 8, wherein the count ofvisible plant-parts-of-interest is determined using a convolutionalneural network.
 12. The system of claim 8, wherein the visibleplant-parts-of-interest comprise bean pods.
 13. The system of claim 8,wherein the first plant and second plants are different varietals ofsoybean plants.
 14. The system of claim 8, wherein the spatial dimensionof the first plant comprises a height of the first plant.
 15. At leastone non-transitory computer-readable medium comprising instructionsthat, when executed by one or more processors, cause the one or moreprocessors to: obtain a plurality of digital images acquired by arange-capable vision sensor carried by a ground-based vehicle past atleast first and second plants, wherein differing spatial dimensions ofthe first and second plants cause parts of the first and second plantsto be first and second distances, respectively, from the range-capablevision sensor as it passes the at least first and second plants; processone or more of the digital images depicting the first plant; based onthe processing: estimate the first distance of the range-capable visionsensor to the first plant, estimate a spatial dimension of the firstplant, and determine a count of visible plant-parts-of-interest of thefirst plant that were captured within a field of view of therange-capable vision sensor; based on the estimated first distance,normalize the count of visible plant-parts-of-interest of the firstplant with another count of visible plant-parts-of-interest determinedfrom one or more of the digital images depicting the second plant; andpredict a crop yield using the normalized count of visibleplant-parts-of-interest and the first spatial dimension of the firstplant.
 16. The at least one non-transitory computer-readable medium ofclaim 15, wherein the estimated first distance of the range-capablevision sensor to the first plant is estimated based on a distribution ofpixel-wise ranges between the range-capable vision sensor and the firstplant, wherein the pixel-wise ranges are detected by the range-capablevision sensor and included in one or more of the digital imagescapturing the first plant.
 17. The at least one non-transitorycomputer-readable medium of claim 16, wherein the estimated firstdistance of the range-capable vision sensor to the first plant iscalculated as a quantile of the distribution of pixel-wise ranges. 18.The at least one non-transitory computer-readable medium of claim 15,wherein the count of visible plant-parts-of-interest is determined usinga convolutional neural network.
 19. The at least one non-transitorycomputer-readable medium of claim 15, wherein the visibleplant-parts-of-interest comprise bean pods.
 20. The at least onenon-transitory computer-readable medium of claim 8, wherein the spatialdimension of the first plant comprises a height of the first plant.